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Growing  I  nterest  in  SE  Effectiveness 

•  Questions  about  the  effectiveness  of  the  SE 
processes  and  activities  are  being  asked 

-  DoD 

-  INCOSE 

-  Others 

•  Key  activities  and  events  have  stimulated 
interest 

-  DoD  SE  Revitalization 

-  AF  Workshop  on  System  Robustness 

•  Questions  raised  included: 

-  How  do  we  show  the  value  of  Systems  Engineering? 

-  How  do  you  know  if  a  program  is  doing  good  systems 
engineering? 

•  Sessions  included  SE  Effectiveness  measures  and  Criteria  for 
Evaluating  the  Goodness  of  Systems  Engineering  on  a 
Program 


Background  of  the  Systems  Engineering 
Leading  I  ndicators  Project 


"SE  Leading  I  ndicators  Action  Team"  formed  in  late  2004 
under  Lean  Aerospace  I  nitiative  (LAI )  Consortium  in 
support  of  Air  Force  SE  Revitalization 

The  team  is  comprised  of  engineering  measurement  experts  from 
industry,  government  and  academia,  involving  a  collaborative 
partnership  with  INCOSE,  PSM,  and  several  others 

•  Co-Leads:  Garry  Roedler,  Lockheed  Martin  &  Donna  Rhodes,  MIT 
ESD/LAI  Research  Group 

•  Leading  SE  and  measurement  experts  from  collaborative  partners 
volunteered  to  serve  on  the  team 

The  team  held  periodic  meetings  and  used  the  I  SO/ 1  EC  15939  and 
PSM  I  nformation  Model  to  define  the  indicators. 

PSM  (Practice  Software  and  Systems  Measurement)  has  developed 
foundational  work  on  measurements  under  government  funding; 
this  effort  uses  the  formats  developed  by  PSM  for  documenting 
the  leading  indicators 
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INCOSE 


International  Council  on  Systems  Engineering 


S \ 'STEM S  E N G I N EERI \TC 
Research  Center 


Systems  Engineering  Advancement  Research  Initiative 


Software  Engineering  Institute 


Carnegie  Mellor 


Nsllonal  Oefonsa  Imfnstf^l  fesocialjm 


...  and  several  others 


4 


Objectives  of  the  project 

1.  Gain  common  understanding  of  the  needs  and  drivers  of  this  initiative 

2.  Identify  information  needs  underlying  the  application  of  SE 
effectiveness 

-  Address  SE  effectiveness  and  key  systems  attributes  for  systems,  SoS, 
and  complex  enterprises,  such  as  robustness,  flexibility,  and  architectural 
integrity 

3.  Identify  set  of  leading  indicators  for  SE  effectiveness 

4.  Define  and  document  measurable  constructs  for  highest  priority 
indicators 

-  I  ncludes  base  and  derived  measures  needed  to  support  each  indicator, 
attributes,  and  interpretation  guidance 

5.  Identify  challenges  for  implementation  of  each  indicator  and 
recommendations  for  managing  implementation 

6.  Establish  recommendations  for  piloting  and  validating  the  new 
indicators  before  broad  use 
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SE  Leading  I  ndicator  Definition 

•  A  measure  for  evaluating  the  effectiveness  of  a  how  a 
specific  SE  activity  is  applied  on  a  program  in  a  manner 
that  provides  information  about  impacts  that  are  likely  to 
affect  the  system  performance  objectives 

-  An  individual  measure  or  collection  of  measures  that  are 

predictive  of  future  system  performance 

•  Predictive  information  (e.g.,  a  trend)  is  provided  before  the 
performance  is  adversely  impacted 

-  Measures  factors  that  may  impact  the  system  engineering 
performance,  not  just  measure  the  system  performance  itself 

-  Aids  leadership  by  providing  insight  to  take  actions  regarding: 

•  Assessment  of  process  effectiveness  and  impacts 

•  Necessary  interventions  and  actions  to  avoid  rework  and  wasted 
effort 

•  Delivering  value  to  customers  and  end  users 
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Leading  I  ndicators 


Sources  of  Smoke  Fire 

ignition  detectors  alarms 


Fires 


Engineering 

Capability 


Causes  + 


Consequences 


Need  to  monitor 
drivers  and  triggers 


Performance 
not  meeting 
plans 


Product  not 
maturing  fast 
enough 


Behind 

schedule, 

unpredictable 
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Criteria  of  Leading  I  ndicators 


•  Early  in  activity  flow 

•  In-process  data 
collection 

•  In  time  to  make  decisions 

-  Actionable 

-  Key  decisions 

•  Objective 

•  Insight  into  goals  / 
obstacles 

•  Able  to  provide  regular 
feedback 


•  Can  support  defined 
checkpoints 

-  Technical  reviews,  etc. 

•  Confidence 

-  Quantitative  (Statistical) 

-  Qualitative 

•  Can  clearly/objectively 
define  decision  criteria 
for  interpretation 

-  Thresholds 

•  Tailorable  or  universal 


Used  criteria  to  prioritize  candidates  for  inclusion  in  guide 


Systems  Engineering  Leading  I  ndicators 


Objective:  Develop  a  set  of  SE  Leading  I  ndicators  to  assess  if  program  is 
performing  SE  effectively,  and  to  enhance  proactive  decision  making 


•Thirteen  leading 
indicators  defined  by 
SE  measurement 
experts 

•  Beta  guide  released 
December  2005  for 
validation 

-  Pilot  programs 
conducted 

-  Workshops  conducted 

-  Survey  conducted 

•  106  responses 

•  Query  of  utility  of 
each  indicator 

•  No  obvious 
candidates  for 

deletion 


•Version  1.0  released  in J  une  2007 
•Version  2.0  released  in  Feb  2010 

-  Enhancements  and  lessons  learned 

-  5  additional  leading  indicators 


SYSTEMS  ENGINEERING  LEADING 
INDICATORS 
GUIDE 

Version  2.0 
January  29,  2010 

Supersedes  Initial  Release,  J  une  2007 


LAIi@> 


iSlCOSE 


INC0SE  Technical  Product  Number:  INCOSE-TP-2005-001-03 
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List  of  I  ndicators  (Original  Set) 


Requirements  Trends  (growth; 
correct  and  complete) 

•  System  Definition  Change 
Backlog  Trends  (cycle  time, 
growth) 

•  I  nterface  T rends  (growth; 
correct  and  complete) 

•  Requirements  Validation  Rate 
Trends  (at  each  level  of 
development) 

•  Requirements  Verification 
Trends  (at  each  level  of 
development) 

•  Work  Product  Approval  T rends 

-  I  nternal  Approval  (approval 
by  program  review  authority) 

-  External  Approval  (approval 
by  the  customer  review 
authority) 


Original  set  had  13  Leading  Indicators 


Review  Action  Closure  T rends 
(plan  vs  actual  for  closure  of 
actions  over  time) 

Technology  Maturity  Trends 
(planned  vs  actual  over  time) 

-  New  Technology  (applicability  to 
programs) 

-  Older  Technology  (obsolesence) 

Risk  Exposure  Trends  (planned 
vs,  actual  over  time) 

Risk  Handling  Trends  (plan  vs, 
actual  for  closure  of  actions  over 
time) 

SE  Staffing  and  Skills  Trends:  # 
of  SE  staff  per  staffing  plan  (level 
or  skill  -  planned  vs.  actual) 

Process  Compliance  Trends 

Technical  Measurement  Trends: 
MOEs  (or  KPPs),  MOPs,  TPMs, 
and  margins 
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List  of  I  ndicators  (added  in  Version  2.0) 

•  Facility  and  Equipment  Availability 
(availability  of  non-personnel 
resources  needed  throughout  the 
project  lifecycle) 

•  Defect  and  Error  Trends  (defect 
discovery  profile  over  time) 

•  System  Affordabi  I  ity  T  rends 
(cost/ effort/ schedule/ performance 
distributions) 

•  Architecture  Trends  (architecture 
process  maturity,  system  definition 
maturity,  architecture  skills) 

•  Schedule  and  Cost  Pressure 
(impact  of  schedule  and  cost 
challenges) 


Version  2  Added  5  Leading  Indicators 
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Fields  of  I  nformation  Collected  for 
Each  I  ndicator 


I  nformation  Need/ Category 

•  Measurable  Concept 

•  Leading  I  nformation 
Description 

•  Base  Measures  Specification 

-  Base  Measures  Description 

-  Measurement  Methods 

-  Units  of  Measure 

•  Entities  and  Attributes 

-  Relevant  Entities  (being 
measured) 

-  Attributes  (of  the  entities) 

•  Derived  Measures  Specification 

-  Derived  Measures  Description 

-  Measurement  Function 


•  I  ndicator  Specification 

-  I  ndicator  Description  and 
Sample 

-  Thresholds  and  Outliers 

-  Decision  Criteria 

-  I  ndicator  I  nterpretation 

•  Additional  Information 

-  Related  SE  Processes 

-  Assumptions 

-  Additional  Analysis  Guidance 

-  I  mplementation 
Considerations 

-  User  of  the  I  nformation 

-  Data  Collection  Procedure 

-  Data  Analysis  Procedure 


Derived  from  measurement  guidance  of  PSM  and  I  SO/ 1  EC  15939,  Measurement  Process 
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Guide  Contents 

1.  About  This  Document 

2.  Executive  Summary 

•  Includes  mapping  of  indicators  to  life 
cycle  phases/ stages 

3.  Leading  Indicators  Descriptions 

•  Description  of  each  indicator  example 
graphics,  and  detailed  definitions  with 
all  fields  of  information 

4.  I  implementation  Considerations 

•  I  ncludes  Cost- Benefit,  Leading 

I  ndicator  Performance,  Composite 
I  ndicators.  Mapping  to  SE  Activities 

5.  References 
Appendices 

•  NAVAI R  Applied  Leading  I  ndicator 
I  implementation 

•  Human  Systems  I  ntegration 
Considerations 

•  Early  Identification  of  SE-Related 
Program  Risks 
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Example  of  Section  3  Contents 


1.1  Requirements  Trends 

This  indicator  is  used  to  evaluate  the  trends  in  the  growth,  change,  completeness  and  correctness  of  the 
definition  of  the  system  requirements.  This  indicator  provides  insight  into  the  rate  of  maturity  of  the 
system  definition  against  the  plan.  Additionally,  it  characterizes  the  stability  and  completeness  of  the 
system  requirements  which  could  potentially  impact  design,  production,  operational  utility,  or  support. 

he  interface  trends  can  also  indicate  risks  of  change  to  and  quality  of  architecture,  design, 
implementation,  verification,  and  validation,  as  well  as  potential  impact  to  cost  and  schedule. 

An  example  of  how  such  an  indicator  might  be  reported  is  show  below.  Refer  to  the  measurement 
information  specification  below  for  the  details  regarding  this  indicator;  the  specification  includes  th< 
general  information  which  would  be  tailored  by  each  organization  to  suit  its  needs  and  organizational 
practices. 


Requirements  Trends 


Requirements  Growth  Trends 


Jan  Feb  Mar  Apr  May  June  July  Aug  Sep 
SRR  PDR  TIME  CDR 


Requirements  Trends.  The  graph  illustrates  growth  trends  in  the  total  number  of  active  requirements 
in  respect  to  planned  number  of  requirements  (which  is  typically  based  on  expected  value  based  on 
historical  information  of  similar  projects  as  well  as  the  nature  of  the  project).  The  measures  shown  could 
apply  to  all  levels  of  abstraction  from  high-level  to  detailed  requirements.  Based  on  actual  data,  a 
projected  number  of  requirements  will  also  be  shown  on  a  graph.  In  this  case,  we  can  see  around  PDR 
that  there  is  a  significant  variance  in  actual  versus  planned  requirements,  indicating  a  growing  problem. 
An  organization  would  then  take  corrective  action  -  where  we  would  expect  to  see  the  actual  growth 
move  back  toward  the  planned  subsequent  to  this  point.  The  requirements  growth  is  an  indicator  of 
potential  impacts  to  cost,  schedule,  and  complexity  of  the  technical  solution.  It  also  indicates  risks  of 
change  to  and  quality  of  architecture,  design,  implementation,  verification,  and  validation. 


Requirements  Volatility:  ABC  Program  - 


Requirements  Volatility.  The  graph  illustrates  the  rate  of  change  of  requirements  over  time.  It  also 
provides  a  profile  of  the  types  of  change  (new,  deleted,  or  revised)  which  allows  root-cause  analysis  of 
the  change  drivers.  By  monitoring  the  requirements  volatility  trend,  the  project  team  is  able  to  predict  the 
readiness  for  the  System  Requirements  Review  (SRR)  milestone.  In  this  example,  the  project  team 
initially  selected  a  calendar  date  to  conduct  the  SRR,  but  in  subsequent  planning  made  the  decision  to 
have  the  SRR  be  event  driven,  resulting  in  a  new  date  for  the  review  wherein  there  could  be  a  successful 
review  outcome. 


TBD/TBR  Discovery  Rate  Curve 


TBEVTBR  Discovery  Rate  Curve 


Cum  TBRs  /  Cum  Month 


Cum  TBfts  f  Cum  Month 


TBD/TBR  Discovery  Rate.  The  graphs  show  the  cumulative  requirement  TBDs/TBRs  vs.  the  ratio  of 
cumulative  TBDs/TBRs  over  cumulative  time.  Each  point  represents  a  successive  instance  in  time  as  you 
move  along  the  graph  from  bottom  to  top.  The  plot  provides  an  indication  of  the  convergence  and  stability 
of  the  TBDs/TBRs  over  the  life  cycle  of  the  project.  The  graph  on  the  left  shows  a  desirable  trend  of 
requirement  TBD/TBR  stability;  as  the  ratio  of  decreases  and  the  cumulative  number  of  TBDs/TBRs 
approaches  a  constant  level.  This  “fold-over”  pattern  is  the  desirable  trend  to  look  for,  especially  in  the 
later  stages  of  project  life  cycle.  In  contrast,  the  graph  on  the  right  shows  an  increasing  number  of 
TBDs/TBRs  even  as  the  project  approaches  later  stages  of  its  life  cycle;  this  is  a  worrisome  trend  in 
system  design  stability.  An  advantage  of  this  plot  is  that,  by  shape  of  the  graph  (without  having  to  read 
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Example  of  Section  3  Contents  (Cont'd) 


Requirements  Trends 

1  nformation  Need  Description 

1  nformation 

Need 

•  Evaluate  the  stability  and  adequacy  of  the  requirements  to  understand 
the  risks  to  other  activities  towards  providing  required  capability,  on- 
time  and  within  budget. 

•  Understand  the  growth,  change,  completeness  and  correctness  of  the 
definition  of  the  system  requirements. 

1  nformation 
Category 

1.  Product  size  and  stability  -  Functional  Size  and  Stability 

2.  Also  may  relate  to  Product  Quality  and  Process  Performance  (relative  to 
effectiveness  and  efficiency  of  validation) 

Measurable  Concept  and  Leading  1  nsight 

Measurable 

Concept 

Is  the  SE  effort  driving  towards  stability  in  the  System  definition  and  size? 

Leading  1  nsight 
Provided 

•  1  ndicates  whether  the  system  definition  is  maturing  as  expected. 

•  1  ndicates  risks  of  change  to  and  quality  of  architecture,  design, 
implementation,  verification,  and  validation. 

•  1  ndicates  schedule  and  cost  risks. 

•  Greater  requirements  growth,  changes,  or  impacts  than  planned  or 
lower  closure  rate  of  TBDs/TBRs  than  planned  indicate  these  risks. 

•  May  indicate  future  need  for  different  level  or  type  of  resources/skills. 

•  1  ndicates  potential  lack  of  understanding  of  stakeholder  requirements 
that  may  lead  to  operational  or  supportability  deficiencies. 

Base  Measure  Specification 

Requirement  TBC 
Requirement  Defe 
Requirement  Cha 
Requirement  Cha 


Requirements  Trends 


Count  the  numbe 


Count  the  numbe 
of  interest;  e.g. 
Priority  Levels,  Q 
Times) 

Estimate  the  imp 


Unit  of 

Measurement 

1.  Requirements 

2.  Requirement  TBDs/TBRs  per  associated 

3.  Requirement  Defects  per  associated  att 

4.  Requirement  Changes  per  associated  at 

5.  Effort  Hours  per  Requirement  Change  (< 
hours  expected  for  each  change) 

Entities  and  Attributes 

Relevant  Entities 

•  Requirements 

Attributes 

•  Requirement  TBDs/TBRs 

•  Requirement  Defects 

•  Requirement  Changes 

•  Additional  attributes  including  but  not  li 
Disposition  Action,  Maturity  States,  Prio 
Classification  Type,  and  Dates  &  Times 
events 

Derived  Measure  Specificat 

Derived  Measure 


Measurement 
Function  * 


Requirements  Trends 


I  ndicator  Specification 


Line  or  bar  graphs  that  show  trends  of  requirements  growth  and  TBD/TBR 
closure  per  plan.  Stacked  bar  graph  that  shows  types,  causes,  and 
impact/severity  of  changes.  Show  thresholds  of  expected  values  based  on 
experiential  data.  Show  key  events  along  the  time  axis  of  the  graphs. 

1.  Line  or  bar  graphs  that  show  growth  of  Requirements  over  time 

2.  Line  or  bar  graphs  that  show  %  Requirements  Approved  over  time 

I  ndicator  3.  Line  or  bar  graphs  that  show  %  TBDs/TBRs  not  closed  per  plan 

Description  and  4.  Line  or  bar  graphs  that  show  %  Requirements  Change 

Sample  5.  Line  or  bar  graphs  that  show  Estimated  I  mpact  of  Requirements  Change 

for  a  given  time  interval  (in  effort  hours) 

6.  Line  or  bar  graphs  that  show  Defect  Profile  (by  types,  causes,  severity, 
etc.) 

7.  Line  or  bar  graphs  that  show  Defect  Density 

8.  Stacked  bar  graph  that  shows  types,  causes,  and  impact/severity  of 


Thresholds  and 
Outliers 

Organization  dependent. 

Decision  Criteria 

Investigate,  and  potentially,  take  corrective  action  when  the  requirements 
growth,  requirements  change  impact,  or  defect  density/distribution  exceeds 
established  thresholds  <fill  in  organization  specific  threshold>  or  a  trend  is 
observed  per  established  guidelines  <fill  in  organizational  specifio. 

I  ndicator 
I  interpretation 


Used  to  understand  the  maturity  of  the  system  definition 
Used  to  understand  impact  on  system  definition  and  impact  on 
production. 

Analyze  this  indicator  for  process  performance  and  other  relationships 
that  may  provide  more  "leading  perspective". 

Ops  Concept  quality  may  be  a  significant  leading  indicator  of  the 
requ 
com 
Care 
drivi 
Reqi 
func 
Revi 


1 .  %  Requ  i  rements  App  roved 

2.  %  Requirements  Growth 

3.  %  TBDs/TBRs  Closure  Variance  per  Plar 

4.  %  Requirements  Modified 

5.  Estimated  I  mpact  of  Requirements  Changes  for  a  given  time  interval  (in 
Effort  Hours) 

6.  Requirement  Defect  Profile 

7.  Requirement  Defect  Density 

8.  Requirement  Defect  Leakage  (or  Escapes) 

9.  Cycle  time  for  Requirement  Changes  (each  and  average) 

1.  (Requirements  Approved  /  Requirements  identified  and  defined)*  100  for 
a  given  time  interval 

2.  ((Requirements  in  current  baseline  -  Requirements  in  previous  baseline) 

/  (Requirements  in  previous  baseline)  *  100 

3.  ((TBDs/TBRs  planned  for  closure  -  TBDs/TBRs  closed)  /  TBDs/TBRs 
planned  for  closure)  *  100 

4.  (Requirements  Modified  /  Total  Requirements)  *  100  for  a  given  time 
interval 

5.  Sum  of  estimated  impacts  of  Requirement  Changes  during  a  given  time 
interval 

6.  Requirement  Defects  for  each  defect  category 

7.  Requirement  Defects  /  Requirements  as  a  function  of  time 

8.  Subset  of  Requirement  Defects  found  in  a  phase  subsequent  to  its 
insertion 

9.  Elapsed  time  (difference  between  start  and  stop  times)  or  total  effort 


Func 


Requirements  Trends 

Additional  1  nformation 

Related 

Processes 

Stakeholder  Requirements,  Requirements  Analysis,  Architectural  Design 

Assumptions 

•  Requirements  Database,  Change  Control  records,  defect  records  are 
maintained  &  current. 

•  TBDs  and  TBRs  are  recorded  and  tracked. 

Additional 

Analysis 

Guidance 

•  May  also  be  helpful  to  track  trends  based  on  severity/priority  of  changes 

•  Defect  leakage  -  identify  the  phases  in  which  defect  was  inserted  and 
found  for  each  defect  recorded. 

1  mplementation 
Considerations 

•  Requirements  that  are  not  at  least  at  the  point  of  a  draft  baseline  should 
not  be  counted. 

•  Usage  is  driven  by  the  correctness  and  stability  of  requirements 
definition. 

o  Lower  stability  means  higher  risk  of  impact  to  other  activities 
and  other  phases,  thus  requiring  more  frequent  review, 
o  Applies  throughout  the  life  cyde,  based  on  risk, 
o  Track  this  information  per  baseline  version  to  track  the  maturity 
of  the  baseline  as  the  system  definition  evolves. 

User  of 

1  nformation 

Data  Collection 
Procedure 

•  Program/ Project  Manager 

•  Chief  Systems  Engineer 

•  Product  Managers 

•  Designers 

•  See  Appendix  F 

Data  Analysis 
Procedure 

•  See  Appendix  F 

Systems  Engineering  Leading  I  ndicators 
Application  to  Life  Cycle  Phases/  Stages 


Table  1  -  SYSTEMS  ENGI NEERI NG  LEADI NG  I NDI  GATORS  OVERVI EW 


Leading 

Indicator 

1  nsight  Provided 

Phases  /  Stages 

P 

1 

P 

2 

P 

3 

P 

4 

P 

5 

S 

1 

s 

2 

S 

3 

S 

4 

S 

5 

Requirements 

Trends 

Rate  of  maturity  of  the  system  definition  against  the  plan. 
Additionally,  characterizes  the  stability  and  completeness  of 
the  system  requirements  which  could  potentially  impact 
design  and  production. 

System 
Definition 
Change  Backlog 
Trend 

Change  request  backlog  which,  when  excessive,  could  have 
adverse  impact  on  the  technical,  cost  and  schedule 
baselines. 

• 

• 

• 

• 

• 

• 

Interface 

Trends 

1  nterface  specification  closure  against  plan.  Lack  of  timely 
closure  could  pose  adverse  impact  to  system  architecture, 
design,  implementation  and/or  V&V  any  of  which  could 
pose  technical,  cost  and  schedule  impact. 

1  Requirements 
Validation 

Trends 

Progress  against  plan  in  assuring  that  the  customer 
requirements  are  valid  and  properly  understood.  Adverse 
trends  would  pose  impacts  to  system  design  activity  with 
corresponding  impacts  to  technical,  cost  &  schedule 
baselines  and  customer  satisfaction. 

1  Requirements 
Verification 
Trends 

Progress  against  plan  in  verifying  that  the  design  meets  the 
specified  requirements.  Adverse  trends  would  indicate 
inadequate  design  and  rework  that  could  impact  technical, 
cost  and  schedule  baselines.  Also,  potential  adverse 
operational  effectiveness  of  the  system. 

Work  Product 
Approval 

Trends 

Adequacy  of  internal  processes  for  the  work  being 
performed  and  also  the  adequacy  of  the  document  review 
process,  both  internal  and  external  to  the  organization. 

High  reject  count  would  suggest  poor  quality  work  or  a 
poor  document  review  process  each  of  which  could  have 
adverse  cost,  schedule  and  customer  satisfaction  impact. 

1  Review  Action 
Closure  Trends 

Responsiveness  of  the  organization  in  closing  post-review 
actions.  Adverse  trends  could  forecast  potential  technical, 
^CQst  and  schedule  baselinejssues. 
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I  ndicator's  Usefulness  for  Gaining  I  nsight  to 


the  Effectiveness  of  Systems  Engineering  (iof2) 


Indicator 

Critic 

al 

Very 

Useful 

Somewhat 

Useful 

Limited 

Usefuln 

ess 

Not  Useful 

Usefulness 
Rating  * 

Requirements  Trends 

24% 

35% 

11% 

3% 

3% 

4.1 

System  Definition  Change  Backlog 
Trend 

7 

11 

7 

3 

1 

3.9 

Interface  Trends 

14 

12 

4 

0 

1 

4.3 

Requirements  Validation  Trends 

22 

16 

4 

0 

1 

4.4 

Requirements  Verification  Trends 

37 

23 

6 

2 

1 

4.4 

Work  Product  Approval  Trends 

7 

19 

21 

2 

0 

3.9 

Review  Action  Closure  Trends 

5 

33 

21 

5 

0 

3.9 

Risk  Exposure  Trends 

14 

37 

6 

1 

0 

4.3 

Risk  Handling  Trends 

6 

25 

11 

1 

0 

4.1 

Technology  Maturity  Trends 

6 

6 

7 

0 

0 

4.1 

Technical  Measurement  Trends 

21 

27 

6 

0 

0 

4.4 

Systems  Engineering  Staffing  & 

Skills  Trends 

11 

27 

15 

0 

0 

4.2 

Process  Compliance  Trends 

6 

14 

11 

1 

0 

4.0 

*  Defined  on  the  Slide  .  □  Somewhat  Useful  □  Very  Useful  Note:  Reflects  Version  1  indicators  only 

Percentages  shown  are  based  on  total  survey  responses.  Not  all  indicator  responses  total  to  100%  due  to  round-off  error  or 

the  fact  that  individual  surveys  did  not  include  responses  for  every  question.  1 8 


I  indicator's  Usefulness  for  Gaining  I  nsight  to 
the  Effectiveness  of  Systems  Engineering  (2of2> 


•  Usefulness  Ratings  defined  via  the  following 
guidelines: 

-  4.6-5.0  =  Critical:  Crucial  in  determining  the  effectiveness 
of  Systems  Engineering 

-  4.0-4.5  =  Very  Useful:  Frequent  insight  and/or  is  very 
useful  for  determining  the  effectiveness  of  Systems 
Engineering 

-  3.0-3.9  =  Somewhat  Useful:  Occasional  insight  into  the 
effectiveness  of  Systems  Engineering 

-  2. 0-2. 9  =  Limited  Usefulness:  Limited  insight  into  the 
effectiveness  of  Systems  Engineering 

-  Less  than  2.0  =  Not  Useful:  No  insight  into  the 
effectiveness  of  Systems  Engineering 
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Additional  I  nformation  on  Specific 
Application  and  Relationships 

1.  Cost-effective  sets  of  Base  Measures  that 
support  greatest  number  of  indicators 

2.  I  ndicators  vs.  SE  Activities  of  I  SO/I  EC  15288 

3.  Application  of  the  SE  Leading  I  ndicators  for 
Human  System  Integration  (HSI) 

4.  Application  of  the  SE  Leading  I  ndicators  for 
Understanding  Complexity 
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SEU  versus  SE  Activities  of  I  SO/ 1  EC  15288 


Requirements  Trends 

System  Definition 

Change  Backlog 

Trend 

Interface  Trends 

Requirements 

Validation  Trends 

Requirements 

Verification  Trends 

Work  Product 

Approval  Trends 

Review  Action 

Closure  Trends 

Risk  Exposure  Trends 

Risk  Handling  Trends 

Technology  Maturity 

Trends 

Technical 

Measurement  Trends 

Systems  Engineering 

Staffing  &  Skills 

Trends 

Process  Compliance 

Trends 

Test  Completeness 

Trends 

i-  -o 

=  c 

> 

Of 

u 

i_ 

3  | 

o 

ut 

£ 

Defect/Error  Trends 

Algorithm/  Scenario 

Trends 

System  Affordability 

Trends 

Architecture  Trends 

6.3  Project  Processes 

6.3.1  Project  Planning  Process 

6.3. 1.3. a  Define  the  project 

6.3. 1. 3. b  Plan  the  project  resources 

X 

X 

6.3. 1. 3. c  Plan  the  project  technical  and  quality  management 

X 

X 

X 

6.3. 1. 3. d  Activate  the  project 

6.3.2  Project  Assessment  and  Control  Process 

6. 3.2.3. a  Assess  the  project 

X 

X 

X 

X 

X 

X 

6. 3.2.3. b  Control  the  project 

X 

X 

X 

X 

X 

X 

6. 3.2.3. c  Close  the  project 

6.3.3  Decision  Management  Process 

6. 3. 3. 3. a  Plan  and  define  decisions 

X 

X 

6.3.3.3.b  Analyze  the  decision  information 

X 

X 

6.3.3.3.C  Track  the  decision 

X 

X 

6.3.4  Risk  Management  Process 

6. 3.4.3. a  Plan  Risk  Management 

6. 3.4. 3. b  Manage  Risk  Profile 

6. 3.4. 3. c  Analyze  Risks 

X 

6. 3.4. 3. d  Treat  Risks 

X 

X 

6. 3.4. 3. e  Monitor  Risks 

X 

X 

6. 3.4. 3. f  Evaluate  Risk  Management  Process 

X 

X 

6.3.5  Configuration  Management  Process 

6. 3. 5. 3. a  Plan  configuration  management 

6.3.5.3.b  Perform  configuration  management 

X 

6.3.6  Information  Management  Process 

6. 3.6.3. a  Plan  information  management 

6.3.6.3.b  Perform  information  management 

X 

6.4  Technical  Processes 

6.4.1  Stakeholder  Requirements  Definition  Process 

6.4. 1.3. a  Elicit  Stakeholder  Requirements 

X 

6.4. 1. 3. b  Define  Stakeholder  Requirements 

X 

X 

X 

6.4. 1. 3. c  Analyze  and  Maintain  Stakeholder  Requirements 

X 

X 

X 

X 

X 

X 
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Leading  I  ndicator  Affinity  Table 


Table  2 


LEADI NG  I NDI CATOR  AFFI  Nl  TY 


Technology  Maturity  (8) 

X 

X 

X 

X 

X 

X 

X 

X 

Technical  Measurement  (6) 

X 

X 

X 

X 

X 

X 

Systems  Engineering 

Staffing  &  Skills  (6) 

X 

X 

X 

X 

X 

X 

Process  Compliance  (3) 

X 

X 

X 

|  Test  Completeness  (11) 

X 

X 

X 

X 

I  x 

! . 

| 

X 

X 

r  i 

x  ! 

X 

X 

X 

Facility  and  Equipment  Availability  (5) 

X 

X 

X 

X 

X 

Defect  and  Error  (6) 

X 

— 

X 

X 

X 

X 

X 

Algorithm/Scenario  (5) 

X 

X 

X 

X 

X 

1  System  Affordabi lity  (5) 

X 

i 

1  1 

! 

|  I 

! 

I 

1  x 

X 

x 

X 

Architecture  (6) 

X 

X 

X 

X 

X 

X 

Schedule  and  Cost  Pressure  (5) 

X 

X 

X 

X 

X 

I  ncluded  in  analysis  of  cost-effective  measures  -  may  support  trade-off  analysis 
of  measures  by  understanding  the  related  measures  22 


NAVAI R  Applied  Leading  Indicators 
(ALI)  Methodology 

•  Systematically  analyzes  multiple  data  elements  for  a 
specific  information  need  to  determine  mathematically 
valid  relationships  with  significant  correlation 

-  These  are  then  identified  as  Applied  Leading  I  ndicators 

•  Provides  a  structured  approach  for: 

-  Validation  of  the  LI  s 

-  Identifying  most  useful  relationships 

•  Unanimous  agreement  to  include  this  in  the  SELI  guide 

•  NAVAI  R  (Greg  Hein)  to  summarize  the  methodology  for 
incorporation  into  the  SELI  Guide  revision  as  an 
appendix 

-  Summary  will  include  links  to  any  supplementary  information 
and  guidance 
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I  interaction  with  SERC  SE  Effectiveness 
Measurement  Project 

•  SE  Leading  I  ndicators  Guide  is  pointed  to  from 
SERC  SE  Effectiveness  Measurement  (EM)  project 
for  quantitative  measurement  perspective 

•  SERC  EM  contribution: 

-  Short-term: 

•  Mapping  of  SE  Effectiveness  Measurement  Framework  to  SE 
Leading  Indicators  (SELI) 

-  51  Criteria  =>  Critical  Success  Factors  =>  Questions  =>  SELI 

❖  Critical  Success  Factors  serve  as  I  nformation  Needs 

❖  Questions  serve  as  Measurable  Concepts 

•  Mapping  of  51  Criteria  to  SELI 

•  Review  to  ensure  consistency  of  concepts  and  terminology 

-  Longer-term: 

•  Work  with  OSD  to  get  infrastructure  in  place  to  support  data 
collection  and  analysis 

-  Tie  to  SRCA  DB  (TBR) 

-  May  require  government  access  and  analysis 
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QUEST1 ONS? 


